A MACHINE LEARNING CLASSIFICATION APPROACH TO DETECT TLS-BASED MALWARE USING ENTROPY-BASED FLOW SET FEATURES

Transport Layer Security (TLS) based malware is one of the most hazardous types, as it relies on encryption to conceal connections. Due complexity TLS traffic decryption, several anomaly-based detection studies have been conducted detect TLS-based using different features and machine learning (ML) algorithms. However, these utilized flow with no feature transformation or relied inefficient transformations like frequency-based periodicity analysis outliers percentage. This paper introduces TLSMalDetect, a approach that integrates periodicity-independent entropy-based set (EFS) generated by technique solve utilization issues in related research. EFS effectiveness was evaluated two ways: (1) comparing them corresponding percentage four importance methods, (2) analyzing classification performance without features. Moreover, new Transmission Control Protocol not explored literature were incorporated into their contribution assessed. study’s results proved number packets sent received superior could remarkably increase up ~42% case Support Vector Machine accuracy. Furthermore, basic features, TLSMalDetect achieved highest accuracy 93.69% Naïve Bayes (NB) among ML algorithms applied. Also, from comparison view, TLSMalDetect’s Random Forest precision 98.99% NB recall 92.91% exceeded best relevant findings previous studies. These comparative demonstrated ability more flows out total malicious than existing works. It also generate actual alerts overall earlier research.Transport